Papers with quantitative and qualitative analysis
Are LLMs Good Annotators for Discourse-level Event Relation Extraction? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks, but their effectiveness over discourse-level event relation extraction tasks remains unexplored. |
| Approach: | They evaluate LLMs' ability to address discourse-level event relation extraction tasks using an open-source model and a commercial model. |
| Outcome: | The proposed model performs poorly on discourse-level event relation extraction tasks. |
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (2024.acl-long)
Copied to clipboard
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, Daniel Fried
| Challenge: | Existing benchmarks focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. |
| Approach: | They propose a benchmark to assess the performance of multimodal web agents . they use visual and textual inputs to process and interpret natural language instructions . |
| Outcome: | a new benchmark assesses the performance of multimodal agents on visually grounded tasks . the benchmark identifies limitations of text-only agents and offers insights towards building stronger agents for the web . |
DEbateNet-mig15:Tracing the 2015 Immigration Debate in Germany Over Time (2020.lrec-1)
Copied to clipboard
Gabriella Lapesa, Andre Blessing, Nico Blokker, Erenay Dayanik, Sebastian Haunss, Jonas Kuhn, Sebastian Padó
| Challenge: | a dataset for germany covering the public debate on immigration is annotated . a political science notion of a claim is used to represent the political discourse . |
| Approach: | They annotate a dataset for german public debate on immigration in 2015 using a political science notion of a claim . they identify claims in newspaper articles, assign them to actors and fine-grained categories and annotize their polarity and date. |
| Outcome: | The dataset is annotated by a political science framework and shows it captures political debate . it shows that political actors can change their positions and take a strong stand against them . |
Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings (2022.acl-long)
Copied to clipboard
Shib Dasgupta, Michael Boratko, Siddhartha Mishra, Shriya Atmakuri, Dhruvesh Patel, Xiang Li, Andrew McCallum
| Challenge: | Word2Box provides a set-theoretic training objective for learning word representations . word representation is not natural, all senses and contexts, levels of abstraction, variants and modifications which the word may represent are forced to be captured by mat t is nunc. |
| Approach: | They propose a fuzzy-set interpretation of box embeddings and learn box representations of words using a set-theoretic training objective. |
| Outcome: | The proposed model improves word similarity tasks on less common words. |
Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolution (2021.findings-acl)
Copied to clipboard
| Challenge: | Using unsupervised entity linking, we solve named entity recognition, coreference resolution and relation extraction tasks together. |
| Approach: | They propose to use a knowledge base to inject information into a joint IE model by using unsupervised entity linking. |
| Outcome: | The proposed model improves on two datasets with 5% F1 score. |
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge (2021.naacl-main)
Copied to clipboard
| Challenge: | Cant is important for understanding advertising, comedies and dogwhistle politics . currently, there are very few resources available for the research of cant . |
| Approach: | They propose a large and diverse dataset for creating and understanding cant from a computational linguistics perspective. |
| Outcome: | The proposed dataset can be used to test word embedding similarity and pretrained language models. |
Adaptively Sparse Transformers (D19-1)
Copied to clipboard
| Challenge: | Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. |
| Approach: | They propose an adaptively sparse Transformer where attention heads have flexible, context-dependent sparsity patterns. |
| Outcome: | The proposed model improves interpretability and head diversity when compared to softmax-based models on machine translation datasets. |
Aspect-based Sentiment Analysis with Type-aware Graph Convolutional Networks and Layer Ensemble (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies only leverage dependency relations without considering their dependency types . a valid and effective approach is demonstrated on six English benchmark datasets . |
| Approach: | They propose to explicitly utilize dependency types for ABSA with type-aware graph convolutional networks . attention is used in T-GCN to distinguish different edges in the graph and attentive layer ensemble to comprehensively learn from different layers of T-gCN. |
| Outcome: | The proposed approach performs well on six English benchmark datasets. |
FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing (2022.acl-long)
Copied to clipboard
| Challenge: | Using pre-trained language models, we evaluate performance group disparities while none of these techniques guarantee fairness, nor consistently mitigate group disparity. |
| Approach: | They present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. |
| Outcome: | The proposed methods show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparity. |
Digital Socrates: Evaluating LLMs through Explanation Critiques (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can provide reasoned explanations, but the nature and quality of those explanations are still poorly understood. |
| Approach: | They propose to define a task of explanation critiquing and train an open-source automatic critique model using this data. |
| Outcome: | The proposed model can provide high-quality, nuanced evaluations without expensive API calls or human annotations. |
Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practices (2021.findings-acl)
Copied to clipboard
| Challenge: | Ethical aspects of research in language technologies have received much attention recently . do we observe a rise in formal ethical reviews of NLP studies? |
| Approach: | They conduct a qualitative and quantitative analysis of the ethics of NLP research . they compare the ethical reviews of NLAs to those of related disciplines . |
| Outcome: | The results compare the ACL Anthology to other related disciplines in the field . the results show that there is a heightened awareness of ethical issues that was previously lacking . |
Identifying and Explaining Discriminative Attributes (D19-1)
Copied to clipboard
| Challenge: | Existing word vector representation models lack latent features (dense vectors) identifying discriminative attributes can motivate the development of word vector models with finer semantics. |
| Approach: | They propose to use a word vector representation model to identify discriminative attributes by combining knowledge graphs with images to construct explicit vector spaces. |
| Outcome: | The proposed model performs comparable to state-of-the-art systems while providing full model transparency and explainability. |
Natural Disaster Tweets Classification Using Multimodal Data (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social media platforms are used for expressing opinions or conveying information. |
| Approach: | They propose a hierarchical system that can integrate multimodal data and perform sequential hierarchic classification. |
| Outcome: | The proposed system can find the damage and its severity along with classify the data into humanitarian categories. |
Benchmarking Multimodal Regex Synthesis with Complex Structures (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets for regex generation from natural language are limited in complexity . Existing regex synthesis datasets are simple and the language used to describe them is not diverse . |
| Approach: | They propose a dataset for regex generation from natural language that generates regexes using a probabilistic grammar and pre-defined macros. |
| Outcome: | The proposed dataset is compared to existing datasets for regex generation from natural language . it generates the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts. |
A Multi-Task Architecture on Relevance-based Neural Query Translation (P19-1)
Copied to clipboard
| Challenge: | Existing models for cross-lingual information retrieval are not aware of the vocabulary distribution of the retrieval corpus. |
| Approach: | They propose a multi-task learning approach to train a Neural Machine Translation model with a Relevance-based Auxiliary Task (RAT) for search query translation. |
| Outcome: | The proposed model achieves 16% improvement over a strong baseline on Italian-English query-document dataset. |
InterIDEAS: Philosophical Intertextuality via LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | a new dataset aims to bridge philosophy, literary studies, and natural language processing (NLP) by integrating theories of intertextuality with bibliometric techniques. |
| Approach: | They propose a dataset that bridges philosophy, literary studies, and natural language processing (NLP) it combines theories of intertextuality from literary studies with bibliometric techniques and recent LLMs . |
| Outcome: | a new dataset bridges philosophy, literary studies, and natural language processing (NLP) to analyze intertextuality . the proposed method helps scholars understand the intellectual, social, and historical relations embedded in texts . it also contributes to the development of language models, authors say . |
How Private are Language Models in Abstractive Summarization? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective protection of private information is essential for knowledge dissemination in sensitive domains such as medical and legal. |
| Approach: | They perform a comprehensive study of privacy risks in LM-based summarization using closed- and four-weight models of different sizes and families. |
| Outcome: | The proposed models show that they leak personally identifiable information in their summaries, compared to human-generated summary summators, which show significantly higher privacy protection levels. |
TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity (2026.findings-acl)
Copied to clipboard
Zheyuan Yang, Liqiang Shang, Junjie Chen, Xun Yang, Chenglong Xu, Bo Yuan, Chenyuan Jiao, Yaoru Sun, Yilun Zhao
| Challenge: | TableVista evaluates multimodal table reasoning under visual and structural complexity . current models struggle to maintain reasoning consistency when structural complexity combined with visually integrated presentations. |
| Approach: | They propose a benchmark for evaluating multimodal table reasoning under visual and structural complexity. |
| Outcome: | The proposed model performs poorly on visual and structural complexity. |